Distributed Expertise: Remote Reference Service on a Metropolitan Area Network

What is the nature of reference work in the digital library? What is the role of the reference librarian where many users serve themselves by means of BIDS and other free at-point-of-use services which emulate the Bath original (e.g. MIDAS in Manchester and EDINA in Edinburgh)? How is the concept of the ‘reference desk’ to be defined where points of presence for both users and librarians are distributed? Can assumptions based on specialist roles and fixed locations migrate to the world of virtual reference work? An Edinburgh-based research project is exploring these issues in the context of the enhanced regional communications now available through EaStMAN (Edinburgh and Stirling Metropolitan Area Network). A major goal is for local HE institutions to explore the possibilities afforded for collaborative reference work. The project team have completed preliminary work with the BIOSIS Abstracts service hosted by the EDINA consortium. Initial analysis has explored the experiences of users and the work patterns of librarians. A pilot interface using World Wide Web has been implemented; the design rationale is based on typologies of user problems and expert response cross a number of genres and modalities of interaction. BACKGROUND The Edinburgh and Stirling Metropolitan Area Network (EaStMAN) links regional higher education institutions (HEIs) on a broadband fibre optic network. Opportunities exist for innovative delivery of services, and novel interactions in teaching and research, and numerous pilot projects are now underway. The project described here was set up to link library reference services at three HEIs (University of Edinburgh, Heriot Watt University and Napier University) with the networked library user community. The overall aim is to advance understanding and practical experience of key aspects in the operation of the ‘virtual’ or ‘digital library’. Whilst this promises users unrivalled access to information, some aspects of library services may prove harder to reproduce or deliver when users cease to visit libraries in person. One example is access to the advice and expertise of trained library staff. The project is an attempt to explore the requirements for the effective provision of such assistance to remotely-located library users. Reference librarians at each HEI have been equipped with networked, multimedia PCs and similar systems placed in public access areas at each site. These are connected to local site Ethernet LANs, with inter-library connectivity provided through EaStMAN. It was envisaged that a mix of delivery media, or genre, would be required in order to provide a match with the various generic situations in which advice might be sought and delivered in a networked environment. These situations can be represented by four combinations of space (same place/different place) and time (same time/different time) parameters reflecting the circumstances in which a consultation might place (see Figure 1). The same place/same time combination expresses the circumstances of the traditional reference consultation, and the other three genres arise out of the networked environment i.e. where participants are: 1. remote from one another, but are communicating within the same time frame (synchronously), 2. remote from one another and communicating in different time frames (asynchrously), or 3. in the same place, but at different times. Figure 1 also shows how the use of the different genre is mapped to these generic consultation situations: • FAQ lists to direct users to information generated from previous enquiries by others which may be relevant to their own problems, • email to post and reply to specific requests for help asynchronously, and • conferencing with desktop workspace sharing facilities for synchronous communication between advice seeker and expert. In some respects, our approach is similar to that of NERD, the Newcastle Electronic Reference Desk, but our aim is to provide support for consultations across a wider range of networked communication genres. A web-based ‘consultation server’ has been set up to provide the platform for the various forms of advice service delivery outlined above, with the added benefit of providing and a common point of entry and user interface. The World Wide Web, with its virtues of machine independence, has quickly become an obvious choice for the implementation of networked information services. For access, users only require a compatible browser, and the rapid spread of web technology spin-offs like Java means that it now offers a complete networked applications development environment. (This may also introduce incompatibility problems for some users.) 1 http://www.ncl.ac.uk/library 2 http://www.remote.lib.ac.uk More important for this project, however, given its relatively short time scale, has been the profusion of public domain web-based tools and components now available for incorporation and adaptation within new applications. An example is AnswerWeb (Ref. 1), which provides a set of tools for creating and maintaining FAQ lists derived from email-based queries, and is being used here to support the FAQ list. Our intention is to support synchronous consultations through the provision of various real-time conferencing tools, including text, audio and video. Until recently, videoconferencing required dedicated (and so expensive) communication links, but with the availability of video cards for PCs, and conferencing software such as CUSeeMe, desktop videoconferencing is now possible (albeit at lower quality) over conventional computer networks, and at relatively modest bandwidths (typically, a desktop videoconference channel requires about 100K bits per second). (It is worth noting, however, that some communication service managers have expressed concern that what has been described as the “undisciplined spread” of desktop videoconferencing will swamp LAN capacity.) OBJECTIVES The project’s long term goal is to explore the requirements for the effective provision of assistance to remotely-located library users. More immediate goals include: 1. exploring the issues raised by remote reference work in a networked help environment, 2. incorporating the findings into a design rationale for a user interface which improves users’ judgments on where to seek what kinds of help, and 3. facilitating the distribution of the scarce resource of human expertise in a cooperative environment. Advances in recent years in the development of on-line services have allowed academic staff and students to access library facilities from their own desktops. Users, freed from the inconvenience of having to visit the library in person, may, however, find it more difficult to consult with library staff when they need assistance. On the evidence of various studies, help of some kind would indeed be beneficial; between 30% and 50% of bibliographic searches result in no ‘hits’ (Ref. 2). Poor results from bibliographic searches may be attributed to various causes, ranging from poor user interfaces, to users’ lack of knowledge of the domain and poor search strategies. Whilst better interface design will undoubtedly alleviate some of these problems, we believe that it cannot entirely displace the role of the reference librarian as a source of expert advice that users may call upon to help solve their search problems. The penalties of not getting timely assistance include: • not finding the appropriate material for learning and research, • finding only part of the information needed, and • wasting time consulting the wrong information sources. The results of our study may have relevance for a wider range of educational applications where exploration of ‘resource-based learning’ is already in evidence and beyond. We focus in this paper on two areas: the role of different media modalities, and the impact of networked reference service on the work patterns of librarians. THE ROLE OF DIFFERENT GENRES A major issue for investigation has been the role of different genres in remote advice services, in the context of a broadband network which can support multi-party audio, video and text communication. The advantages (and disadvantages) of both distributed work and varieties of mediation have been evaluated in libraries (Ref. 3; Ref. 4) and other contexts (e.g. call centres, travel agents) and we have drawn on some of these findings in our description of librarians at work (Ref. 5; Ref 6). Recently, genre theory has been used to evaluate the efficacy of different modes of interaction on the World Wide Web (Ref. 7). As we explain below, this approach has informed our design rationale. Traditionally reference work has implied face-to-face meeting between expert and user where the results of searching moves are evaluated and fine-tuned (Ref. 8). For the design of a remote reference consultation service, the question is to what extent the conditions of face-to-face interaction i.e. synchronous communication with visual contact and a shared workspace or context need be recreated for it to be effective. Face-to-face communication is able to draw upon a rich variety of verbal and non-verbal contextual cues which make a significant contribution to communication structure and management. Videoconferencing with desktop workspace sharing between participants is the closest achievable (excluding virtual reality encounters) recreation of face-to-face interaction, but there are alternative forms of service delivery which may also come into their own in certain circumstances. Asynchronous forms of interaction include the posting of queries by email (already established in many academic libraries in the case of the EDINA service, bibliographic database users have spontaneously started sending email queries), or the searching of public FAQs lists. The latter is particularly valuable in that it provides the means to accumulate information ‘capital’ and to make it available to the wider user community. These various forms of interaction have different characteristics and it is important for both overall service quality and its cost-effectiveness to determine the circumstances in which the various options would be most appropriate for both expert and user. Three characteristics are especially relevant here: 1. problem solving capacity i.e. the extent to which they can easily support the presentation and resolution of difficult problems, 2. responsiveness i.e. the speed with which an answer is forthcoming, and 3. availability i.e. the accessibility of the service. We can summarise the service options in terms of these characteristics as shown in the table below. Synchronous conferencing, as a surrogate for face-to-face interaction, scores highly for problem solving capacity and responsiveness, but its availability cannot be guaranteed. Email sacrifices the robustness and communicative power of face-to-face interaction and so scores only moderately well for problem solving capacity, but places no restrictions on a party’s freedom to initiate a consultation, though the response time may be unpredictable. Finally, FAQ lists score lowest for problem solving capacity, since a solution essentially depends on finding a match with a previously posted query, but are without restriction on their responsiveness and availability. Problem solving capacity Responsiveness Availability Conference High High Low Email Medium Medium High FAQ list Low High High Table 1: Genre options and their characteristics. At the present time, we do not know what specific genres, or systems of genres, will emerge as alternatives to the classic reference set. As we shall see later, some notion of how librarians and users might respond can be gleaned from our investigation of the former’s current work practices, and the latter’s current patterns of advice seeking behaviour. As experience is gained from running the pilot consultation service, we intend to elicit more information on how the demands of a given medium are perceived by both librarians and users, how judgements are made by the former on issues such as the prioritisation of tasks, and how the latter perceive its effectiveness for solving their problems. At the beginning, when understanding of the relevance and consequences of these characteristics is limited, patterns of use may be quite unpredictable, but we can expect to see an ‘ecology’ of genre use evolving over time as both experts and users learn to match genres against their particular constraints and requirements. For example, librarians and users may find that specific systems of genres make for an effective solution: consultations may be initiated synchronously, and then pursued to a conclusion asynchronously, and vice versa. PROFESSIONAL WORK PATTERNS A project on networked reference consultation creates the opportunity to explore ripple effects (positive and negative) on other aspects of library service management. Within each of the participating HEIs, local subject and reference librarians’ time is in short supply relative to growing demand. By allowing expertise to be distributed across conventional institutional boundaries, the project could help in the management of this demand and improve service by enabling staff at the participating institutions to pool their time and expertise more effectively. A shared reference advice service available to any library user in any of the participating institutions would extend access to expertise, but with an expert in a given institution ‘on call’ for only a fraction of those hours, and thus free to attend to other duties without jeopardising client interests. This is the positive side. On the downside, library staff may have reservations, including: • worries about their competence to deal with the new technology, • alarm about the possible intrusiveness of some forms of conferencing (e.g. video-), and • the project’s possible longer term implications for employment. Implications at the institutional level may also be raised: how can collaboration be reconciled with competitive trends in Higher Education? Will a distributed service lead to traditional boundaries becoming blurred? Who will then: • determine responsibilities for service management? • lay down guidelines on where loyalties should lie does the consortium or the institution that pays an expert’s salary have priority? Management and ownership issues in a networked environment have only recently emerged as agenda issues in networked education in the United Kingdom (Ref. 9): there are few templates or guides to best practice. We decided to forego a detailed exploration of managerial issues until the pilot service was established, and focus on understanding reference consultations at the micro level in the participating sites, by surveying and observing both users and librarians. THE INVESTIGATION Over a period of six months, we worked with three groups: reference librarians in three institutions where BIOSIS Abstracts, via the EDINA consortium, is used to support course work and research; end-user populations; and the EDINA user group, a UK-wide group of librarians evaluating the beta version of a dedicated BIOSIS interface designed for free-at-point-of-use access in UK academic institutions. We have used a number of techniques to gather data, including interviewing, diary keeping, videoing ‘peer’ reference advice consultations between librarians, and direct observation. THE USERS’ EXPERIENCE A user questionnaire was designed to help build up a broad picture of usage, users’ views of the BIOSIS system and to develop a picture of the kinds of problems that were typically encountered; a total of 38 questionnaires was returned. The largest group of respondents were academic staff (30%), followed by postgraduate students (24%), undergraduate students (20%), research staff (14%) and finally miscellaneous library and information services staff (12%). We present below some initial findings: a more detailed analysis can be found in Procter et al., 1997a (Ref. 10). Users were asked how successful they thought they had been when using the IR system. The approximate breakdown of replies was as follows: Unable to find anything 2% Found some useful information 36% Found what they wanted 44% Users were also asked questions regarding their competence in performing searches. The majority of respondents 61% stated that they judged themselves as either competent or very competent at performing complex searches; however, a significant minority -39% replied that they did not consider themselves competent in this respect. The majority of undergraduates placed themselves in this latter group. Problems reported ranged from the technically trivial but practically significant (e.g. forgotten passwords), to procedural (e.g. command language syntax) and conceptual problems (e.g. compound searches). The final part of the questionnaire was designed to gather users’ opinions and expectations of the different ways of dealing with the problems they encounter when using the IR system. Specifically, users were asked to rate various combinations of sources of assistance and communication genres in terms of: 1. their effectiveness in providing assistance and 2. their convenience i.e. their timeliness and ease of access. Consultation with domain experts scored most highly for effectiveness across all genres: face-to-face, telephone, email. There were, however, interesting differences in users’ perceptions of convenience: of the three communication modalities, email was the most highly rated, and the telephone the least highly rated, with face-to-face coming somewhere in between. The explanation for this may lie in users’ experience of the work patterns of the experts: as we have seen, these mean that experts’ availability may be a very hit and miss affair. Some differences were also found across user categories. For example, academic staff rated the convenience of all forms of unmediated interaction as less convenient than did students. Of the mediated interaction forms, academic staff rated the telephone as more convenient than students, but rated email as less convenient. In contrast, ratings of effectiveness were broadly similar. The findings suggest that a ‘space/time’ factor is at work: where users work from a dedicated space (their own office desk, as is the case for most academics) they do not wish to move and mediated interaction is preferred. Where users are nomadic (the case of most of the undergraduates) they will go to the library. LIBRARIANS AND LIBRARY WORK In our observations of librarians we took a participative design approach (Ref. 11) in order to encourage development of an on-going dialogue between developers and reference librarians and enhance the adaptiveness of the prototype. Logs kept by participants (for one complete week at different stages in the investigation) reveal a pattern of interaction which is characterised by high volume, impersonal, uninformed, short exchanges. Most of the collaborations logged in diary records were short (3 to 5 minutes) interactions of the ‘help desk’ kind, rather than subject searches, and followed a pattern of multiple demands: this may suggest a shift in the nature of reference work from the provision of domain-specific answers to the provision of support (Ref. 12; Ref. 13) . BIOSIS consultations are only a small part of the work remit of these professionals. This asymmetry (what is a totally urgent request to a BIOSIS user unable to proceed may not be a top priority to an expert) may explain some of the comments on ‘convenience’ in the user survey returns. The dearth of sustained ‘problem-solving’ consultations that characterise classic reference work (Ref. 14) reflects the fact that demands for this type of interaction are less frequent in a world where end-users tend to serve themselves. The rise of autonomous searching by end-users has been mapped in a series of UK longitudinal studies by East and his colleagues (Ref. 15). The comments of East's interviewees are echoed in those of two librarians in our study who stated that after induction or training sessions an a given database, users just disappear: “we don't know where they go”. The picture emerges of a community whose members (both librarians and users) are increasingly anonymous, a situation which if not directly attributable to the growth of digital information services, is at least exacerbated by them. SITUATED ACTION Location proved to be a central issue for librarians (as it was for users). Librarians cope with multiple demands by being both flexible and mobile and operating at a number of service points. The most consistent point of presence was, in all cases, a general reference desk, (one workstation for the pilot was sited in this location), where librarians multi-task or parallel process at least some of the time (answering a phone query, for example, while working with paper, or dealing with email; cataloguing, online searching and dealing with verbal enquiries at the reference desk), with the result that ground has to be cleared before the ‘moves’of online searching (Ref. 16) are brought into play. One reference librarian did not have exclusive access to a room, and claimed that the concept of even a desk was difficult to entertain. In addition to the ‘cold calling’ or ‘starting over’ discussed above, ‘hot desking’ is a feature of working life for some of the practitioners in the study, who share office space on an ad hoc basis and often function as nomadic workers. The participating librarians appear to apply criteria to structure their handling of complex demands. Like Suchman's ‘situated’ office workers (Ref. 17), they derive rules of procedure as technology interacts with practices in an evolving work environment. Some rules, for example, appear to relate to priority: in a mediated interaction, visual demands may have precedence over audio, and audio has precedence over text. According to one librarian, video, as a near equivalent of faceto-face, would have priority over phone and email; others indicated that access to video consultation (a ‘demanding’ medium like face-to-face) should be managed, like face-to-face interaction, by means of a timetable of ‘office hours’ or ‘desk hours’. Some rules to accommodate multiple tasks in limited time appear to be based on a hierarchy of perceived user needs ranked on a continuum from ‘domain’ to ‘janitorial’ highly salaried activities have priority. Requests for assistance for what are perceived as janitorial tasks (requests to load paper into the printer or photocopier) are refused at some help desks. But help with loading CD ROM disks, and log-ins for online searches, is readily given as these relate to ‘professional’ skills, as does help with the formulation of search strategies. We have developed the theme of the ‘situated intermediary’ elsewhere (Ref. 18). MATCHING USERS’ PROBLEMS AND EXPERTS’ RESPONSES The specification for the pilot interface and service design accommodates our findings from recent work with users and experts. An immediate issue is the ‘convenience’ factor, or ways in which experts and users may be brought together more effectively, and we have focused on two areas: ‘space/time’ and ‘mutual understanding’. Overall, our findings from user and librarian groups indicate that timing of response and location of interaction are important factors convenience rankings. This suggests that modalities may be tuned to meet location criteria. The asynchronous characteristics of email, for example, make it an obvious genre for a nomadic user linking with a nomadic expert. In a networked environment where reference work involves multiple ‘starting overs’ between agents who are not known to each other, rather than sustained 'problem solving' with familiar clients, what mechanisms can be brought into play that allow reference librarians to ‘fast track’ into an appropriate relationship with the client? User profiling may be of little general use unless it relates to the task in hand: “Knowledge of user status and formal educational background is in general rather uninteresting, superfluous and unusable for inferring anything reasonable by an intermediary in an actual situation” (Ref. 19). Current IR research has focused on the recent history of a client within system space as a proper object of study; two approaches are particularly relevant to our study. Twidale and his colleagues propose visualisation of ‘instant history’ (Ref. 20) as a key to fast profiling, and Campbell (Ref. 21) suggests ‘ostensive’ retrieval (where each step in a search is taken as an indication of level of satisfaction at that stage) as a media and context-free retrieval mechanism: each of these approaches may greatly accelerate both user and expert understanding of what sequence of actions has led to a problematic position. User histories may be saved to provide archives of cumulated individual behavior (just like FAQ archives). In this way, users and experts can “learn appropriate behavior” (a phrase that is common in situated learning) and pick up cues which can inform subsequent moves within a reference interaction (Ref. 22). To explore how mutual understanding might be improved within a networked reference consultation system, we have drawn on work on genre theory (Ref. 23; Ref. 24; Ref. 25), which has allowed us to propose a taxonomy of appropriate actions in appropriate settings. This taxonomy can be supported by different interaction modalities: email, interview, conference etc. and a primitive prototype of an interface is presented in Figure 2. (Each of the icons for modality may mask further icons that relate to specific genres within modalities.) As sequences of interaction become more clearly defined by the service-in-use, windows for additional genres will be added (search history or user moves; on-hold). For further details of our use of genre theory for design rationale, see Procter et. a l 1997b (Ref. 26). Distributed expertise is not a new topic in information science, and some of the areas which we have chosen to articulate tentatively as genres have been selected as structural elements in earlier work on the design of intelligent retrieval systems. The INSTRAT model, for example, of distributed problem treatment (DTP) proposed by Belkin et al., 1983 (Ref. 27) and later developed in MONSTRAT, is highly relevant to our project. Elements of interaction in this system are classified in terms of objectives and responsibilities, and the latter are spread across both machine and human being. Equally relevant to our pilot are the elements specified by Ingwersen (Ref. 28) in his proposed MEDIATOR interactive retrieval engine, as he emphasises the importance of mutual understanding, and the importance of support as well as domain knowledge. MEDIATOR is proposed as an intelligent replacement for human help. Figure 2: Pilot text conferencing tool. The end-product which we envisage differs from MEDIATOR as the system will offer intelligent enhancement of human help: the situated end-user and the situated intermediary will respond to cues from each other, mediated more or less by a guiding interface (Ref. 29), which can indicate what types of assistance are available. The distribution of human expertise in our system, will be both spatial and functional spread across institutions, and across professional and non-professional experts, and across an activity range which mixes both humans and machine-bound agents and covers both advice on what to do and advice on where to seek advice. CONCLUSIONS AND FUTURE DEVELOPMENTS If mixed demands and anonymous transactions are likely to be characteristic of the digital reference economy, users and librarians require an improved understanding of the reference resource (Ref. 30). The taxonomic approach described in this paper may contribute to such understanding. Although it has been derived under local circumstances, it may have value as a generic approach to user interface design in other networked reference support environments. We are planning to explore the transferability of our proposed design rationale in two ways: 1. across databases by analysing email helpline transactions for a number of services in Edinburgh and elsewhere, to establish whether ‘genre repertoires’ can be observed which may be translated into screen ‘cues’. 2. across institutions by working on joint bids to explore a common reference space by means of the EC the Telematics programme with institutons outside the EaSTMAN metropolitan area network. Finally, we will continue to investigate the management and ownershop issues that arise within the networked service emvironment. ACKNOWLEDGMENTS The authors wish to acknowledge the work of Lorna Banks and Kenny McDonald, who set up the technical platform, and collected data. They also wish to thank the participant librarians, and Sheila Cannell of Edinburgh University Library and Peter Burnhill, Director of EDINA, whose support has been essential to the study. The comments of anonymous referees have also been helpful. The project was funded by the Scottish Higher Education Funding Council. Ana Goldenberg is sponsored by CNPq (Brazilian National Research Council). CONTACTFurther information can be obtained from Dr Rob Procter, Dept of Computer Science,James Clerk Maxwell Building, University of Edinburgh. His email [email protected] and his fax is 0131 667 7209. REFERENCES1. Ackerman, M. and McDonald, D. (1996) Answer Garden 2: MergingOrganisational Memory with Collaborative Help, Proceedings of CSCW’96,Boston, ACM Press, 97-105.2. Marchionini, G. (1995) Information Seeking in Electronic Environments,Cambridge University Press, Cambridge.3. Abels, E.G. and Liebscher, P. (1994) A new challenge for intermediary-clientcommunication: the electronic network. The Reference Librarian, 41/42, 185-196.Bristow, A. and Buechley, M. (1995) Academic reference service over email: anupdate. C & RL News, July/August, 459-462.4. Rouncefield, M. et al. (1994). Working with Constant interruption : CSCW and thesmall office. In Proceedings of the ACM 1994 Conference on cooperative work.Chapel Hill, North Carolina, October, ACM Press, 275-286.5. Rogers, Y. (1994) Exploring obstacles: integrating CSCW in evolvingorganisations. In Proceedings of the ACM 1994 Conference on cooperative work.Chapel Hill, North Carolina, October, ACM, New York, 67-77.6. Erickson, T. (1997) Social interaction on the Net: virtual community asparticipatory genre. In R.H. Sprague (ed.) Proceedings of the thirtieth HawaiiInternational Conference on System Sciences, Volume VI, Washington: IEEEComputer Society Press, 13-21.7. Fidel, R. (1985) Moves in online searching. Online Review, 9, 61-74.8. Buckner, K. and Davenport, E. (1996). Support issues for case-based learning in anundergraduate human factors class. Paper presented at the NetLinks Symposium onNetworked Learner Support, University of Sheffield, June.9. Procter, R., Davenport, E., McKinlay, A. and Goldenberg, A. (1997) Supportingcollaborative information retrieval in the virtual library. Paper presented at theMultimedia Minded Conference, Lanaken, Belgium, April 10-12.10.Greenbaum, J. and Kyng, M., eds. (1991) Design at work: cooperative design ofcomputer systems, Lawrence Erlbaum, Hillsdale, NJ.11.Ingwersen, P. (1994) Information Retrieval Interaction. London; Taylor Graham,182-183.12.Brajnik, G., Mizzaro, S. and Tasso, C. (1996) Evaluating user interfaces toinformation retrieval systems: a case study on user support. In SIGIR 96Proceedings, ACM Press, 128-136.13.Taylor, R.S. (1968) Question-negotiation and information seeking in libraries.College and Research Libraries, 178-194.14.East, H., Sheppard, E. and Yeal, Y. (1996) A huge leap forward a quantitativeand qualitative examination of the development of access to database services byBritish universities 1988-1994. CCIS Policy Paper No 5. University ofWestminster: CCIS, May.15.Fidel. op. cit.16.Suchman, L. (1987) Plans and situated actions: the problem of human-machinecommunication. Cambridge: Cambridge University Press.17.Davenport, E. and Procter, R. (1997) The situated intermediary: remote advicegiving in a distributed reference environment. In M.Williams (ed.) Proceedings ofthe National Online Meeting, New York, 12-14 May.18.Ingwersen, P. op. cit., p, 181. 19.Twidale, M. and Nichols, D. (1996) Interfaces to support collaborative informationretrieval, In C.W. Johnson and M. Dunlop (eds.) Workshop on InformationRetrieval and Human Computer Interaction (GIST Technical Report G96-2), 25-28.20.Campbell, I. Applying ostensive functionalism. (1996) In C.W. Johnson and M.Dunlop (eds.). Workshop on Information Retrieval and Human ComputerInteraction (GIST Technical Report G96-2), Glasgow University, 78-82.21.Fleming, W.G. (1994) Methodography: the study of student learning as situatedaction. In Gibbs, G. (ed.) Learning approaches evaluation and strategy. ImprovingStudent learning through assessment and evaluation, Oxford: Oxford BrookesUniversity, 525-544.22.Orlikowski, W.J. and Yates, J. (1994) Genre repertoire: norms and forms forwork and interaction. MIT Sloan School Working Paper 3671-94; Centre forCoordination Science Technical Report 166, March. MIT, Cambridge MA.23.Berkenkotter, C. and Huckin, T.N. (1993) Rethinking genre from a sociocognitiveperspective. Written Communication, 10 (4), 475-509.24.Crowston, K. and Williams, M. (1997) Reproduced and emergent genres ofcommunication on the World-Wide Web. In Sprague, R.H., Editor, Proceedings ofthe thirtieth Hawaii International Conference on System Sciences, Volume VI,Washington: IEEE Computer Society Press, 30-39.25.Procter, R. Goldenberg, A., Davenport, E., Burnhill, P. and Cannell, S. (1997)Genres in support of collaborative information retrieval in the virtual library.Interacting with Computers. (Submitted)26.Belkin, N. et al. (1983) Distributed expert problem treatment as a model forinformation system analysis and design. Journal of Information Science, 5, 153-187.27.Ingwersen, P. op. cit. Chapter 8, pages 203-222.28.Brajnik et al. op. cit.29.Fine, S. (1995) Reference and resources: the human side. Journal of AcademicLibrarianship, January, 17 ff.